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INTRODUCTION 

The  littorals  of  the  World  are  areas  of  great  strategic  importance.  Approximately  95%  of 
the  world’s  population  lives  within  600  miles  of  sea,  80%  of  all  countries  border  the  coast  and 
80%  of  the  world’s  capitals  lie  within  300  nautical  miles  of  shore.  With  the  end  of  the  cold  war, 
DoD’s  focus  has  shifted  Irom  a  large  land/sea  battle  scenario  with  a  monolithic  global  adversary 
to  dealing  with  low-intensity  conflicts  in  the  near-coastal  or  littoral  regions.  In  wartime,  the 
ability  to  dominate  the  littoral,  including  the  imdersea  environment,  allows  operating  with 
impunity  in  the  face  of  area  denial  threats  while  taking  initial  action  to  defeat  those  threats  and 
prepare  the  battlespace  for  follow-on  forces. 

Information  superiority  -  understanding  and  exploitation  of  the  natural  environment  —  is 
critical  to  safe  and  effective  operation  of  joint  forces  engaged  in  littoral  expeditionary  warfare. 
The  tactical  advantage  will  probably  depend  not  on  who  has  the  most  expensive,  sophisticated 
platforms  -  but  rather  in  who  can  most  fully  exploit  the  natural  advantages  gained  by  a  thorough 
understanding  of  the  physical  environment.  Every  aspect  of  the  littoral  environment  is  of  critical 
import  to  conducting  military  operations,  a  huge  challenge  to  both  the  warfighters  and  the 
Meteorology  and  Oceanography  (METOC)  conununity  that  supports  them. 

Characterization  of  the  battlespace  environment  to  ensure  optimal  emplo)anent  of 
personnel,  systems  and  platforms  depends  on  continuous  improvement  in  science  and  technology 
toward  that  aim.  Key  and  essential  elements  in  that  characterization  are  effective,  relocatable, 
high-resolution,  dynamically  linked  mesoscale,  meteorological,  coastal  oceanographic,  surface 
water/groimdwater,  riverine/estuarine  and  sediment  transport  models.  The  linkages  between  these 
models,  coupled  with  the  complexity  and  the  scale  of  the  environment,  require  the 
implementation  of  focused  sets  of  these  models  on  DoD  High  Performance  Computing  (HPC) 
architectures  in  support  of  real  time  operations,  scenario  (operational  and/or  training)  planning, 
and  coarse  of  action  analysis.  Figure  1  shows  the  wide  range  of  physical  processes  that  occur  in 
littoral  waters. 

The  High  Fidelity  Simulations  of  Littoral  Environments  (HFSoLE)  Portfolio  is  funded  by 
CHSSI  under  the  HPCMO.  The  HFSoLE  effort  is  broken  into  four  projects.  1)  The  System 
Integration  and  Simulation  Framework  Project  is  developing  the  interface  algorithms  for  the 
atmospheric,  riverine,  estuarine,  wave,  tidal  and  ocean  models  in  this  portfolio.  2)  The  Near¬ 
shore  and  Estuarine  Environments  Project  is  coupling  and  enhancing  nearshore  process  models  to 
efficiently  simulate  interactions  between  winds,  waves,  currents,  water  levels,  and  sediment 
transport.  3)  The  Surface  Water/  Groundwater,  Riverine  and  Tidal  Environments  Project  is 
enhancing  the  capabilities  of  the  Adaptive  Hydraulics  (ADH)  model.  This  project  is  working 
toward  providing  DoD  the  tools  to  generate  fast  and  accurate  estimates  of  water  depths  and 
velocity  distributions  for  use  in  ingress/egress  assessment  for  rivers  and  streams,  flow  rate  and 
sediment  loading  as  input  for  coastal  simulation,  and  ground-surface  inundation  and  moisture 


content  predictions  for  use  in  local  atmospheric  simulations.  4)  The  System  Applications  Project 
is  improving  the  fidelity  and  execution  speed  of  DoD  maritime  operation  simulation  systems 
dependent  on  data  from  ocean  surface  behavior  models.  Such  simulations  include  amphibious 
assault,  mine  clearing  and  sea  keeping  operations.  This  project  provides  for  the  development, 
implementation  and  testing  of  an  environmental  server  to  support  Modeling  &  Simulation 
(M&S). 


Figure  1.  Physical  processes  that  occur  in  littoral  waters  include  wind  forcing,  tides,  waves,  ocean 
circulation  and  sediment  transport. 

Models  And  Applications 

The  following  section  presents  an  overview  of  the  ocean  circulation,  nearshore  process, 
and  surface  water/groundwater  models  contained  in  HFSoLE.  Grand  challenge  time  was  utilized 
in  performing  model  simulations  with  the  goal  to  support  DoD  efforts. 

HYCOM 

A  series  of  ocean  circulation  simulations  in  the  Japan/East  Sea  have  been  performed  on 
the  Cray  T3E  (NAVO  MSRC)  under  the  HPC  HFSoLE  Grand  Challenge  Project.  The  model  used 
is  the  Hybrid  Coordinate  Ocean  Model  (HYCOM),  a  next  generation  circulation  model  under 
development  at  the  Naval  Research  Laboratory.  Ideally,  an  ocean  general  circulation  model 
should  (a)  retain  its  water  mass  characteristics  for  centuries  (a  characteristic  of  isopycnal 
coordinates),  (b)  have  high  vertical  resolution  in  the  surface  mixed  layer  (a  characteristic  of  z- 
level  coordinates),  and  (c)  have  high  vertical  resolution  in  surface  and  bottom  boimdary  layers  in 
coastal  regions  (a  characteristic  of  terrain-following  coordinates).  Hence,  HYCOM  is  designed  to 
be  isopycnal  in  the  open,  stratified  ocean,  but  makes  a  dynamically  smooth  transition  to  a  terrain¬ 
following  coordinate  in  shallow  coastal  regions,  and  to  z-level  coordinates  in  the  mixed  layer  and 
unstratified  seas. 

The  Japan/East  Sea  has  been  used  to  test  the  robustness  of  HYCOM  because  it  contains 
many  circulation  features  found  in  the  global  ocean,  such  as  a  western  boundary  current,  subpolar 
gyre,  and  rich  eddy  field,  and  it  is  small  enough  to  allow  simulations  with  very  high  resolution. 
Under  this  grand  Challenge  Project,  simulations  with  3.5  km  horizontal  resolution  were 


performed.  Scalability  was  via  MPI/SHMEM,  or  OpenMP,  or  both,  although  SHMEM  was  used 
exclusively  for  the  simulations  performed  on  the  T3E.  The  mesh  size  for  the  high-resolution 
configuration  is  394x618x15,  and  a  one-year  simulation  requires  about  12,000  CPU  hours  per 
model  year. 

High  horizontal  grid  resolutions  are  needed  to  adequately  resolve  the  mesoscale  features 
as  well  as  the  coastline  and  bottom  topography,  especially  over  the  shelf  The  impact  of 
progressively  increasing  the  grid  resolution  is  shown  in  Figure  2.  At  14  km  resolution,  unrealistic 
overshot  of  the  East  Korean  Warm  Current  (140  °E,  39  °N)  is  significant.  This  overshoot  is 
diminished  at  7.5  km,  and  at  3.5  km  the  current  separates  from  the  coast  at  the  observed  latitude. 
Also,  the  strength  of  the  subpolar  gyre  strengthens  as  the  horizontal  resolution  is  increased. 
Simulations  with  the  Navy  Layered  Ocean  Model  (NLOM)  suggest  that  these  changes  in  surface 
circulation  are  due  to  interaction  with  abyssal  currents  that  are  influenced  by  the  bottom 
topography.  The  same  mechanism  appears  to  be  dominant  here,  but  further  analysis  is  needed  to 
be  sure. 


Figure  2.  Impact  of  increasing  the  horizontal  resolution  of  the  HYCOM  model  in  the  Japan/East 
Sea. 


NCOM 


Coastal  regions  will  be  included  in  the  Navy  Coastal  Ocean  Model  (NCOM)-based  global 
nowcast/forecast  system  which  will  serve  as  a  complementary  model  in  the  Hybrid  Coordinate 
Ocean  Model  (HYCOM)-based  global  nowcast/forecast  system.  Global  NCOM  employs  a 
sigma-z  coordinate  system  (Martin,  2000)  in  a  fully  global  domain  including  the  Arctic  and 
continental  shelves.  This  will  provide  high  vertical  resolution  in  coastal  regions  while 
maintaining  a  specified  minimum  vertical  resolution  in  the  deep  ocean  with  a  non-uniform 
distribution  of  levels  to  maintain  higher  near-surface  resolution(Hurlburt,  Barron,  and  Wallcraft, 
personal  communication). 

Global  NCOM  has  1/8®  resolution  at  mid-latitudes  in  a  2048x1280x41  curvilinear  sigma- 
z  grid  with  19  sigma  levels  and  maximum  surface  level  thickness  of  Im  (Martin,  2000).  It  is 


forced  by  the  Navy  Operational  Global  Atmospheric  Prediction  System  (NOGAPS  4.0)  (Hogan 
and  Rosmond,  1991)  from  Fleet  Numerical  Meteorology  and  Oceanography  Center  (FNMOC). 
Daily  2-D  and  3-D  temperature  and  salinity  fields  produced  by  flie  Modular  Ocean  Data 
Assimilation  System  (MODAS)  (Fox  et  al,  2002)  at  the  Naval  Oceanographic  Office  are 
assimilated.  MODAS  uses  climatological  correlations  between  steric  surface  height  (SSH) 
deviation,  surface  temperature  (SST)  deviation  and  subsurface  temperature  deviation  to  predict  3- 
d  temperature  fields  from  surface  information.  SST  from  the  operational  MODAS2D  analysis  of 
satellite  SST  and  SSH  from  the  operational,  altimetry-assimilating  1/16®  NLOM  (Wallcraft, 
1991)  provide  input  for  the  3D  MODAS  calculations.  MODAS  routines  analyze  the  NCOM 
temperature  fields  to  determine  mixed  layer  depth,  which  is  defined  as  the  depth  (if  any)  at  which 
temperature  drops  to  0.25  "C  below  the  surface  temperature  (Rhodes  et  al.,  2002). 

1/8°  global  NCOM  model  simulations  have  been  performed  on  the  NAVO  IBM-SP  on 
128  processors  (32  nodes)  with  a  wallclock  limit  of  two  hours.  Each  model  simulation 
corresponds  to  10  model  days.  This  can  be  performed  in  reanalysis  mode,  hindcast  mode,  or  a 
daily  mode  consisting  of  a  hindcast  of  three  to  five  days,  an  analysis,  and  then  a  forecast  of  three 
days.  Figure  3  depicts  a  recent  Global  NCOM  sea  surface  temperature  analysis. 
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Figure  3.  NCOM  sea  surface  temperature  analysis  for  June  5, 2002. 

STWAVE 

Ocean  waves  near  the  coast  have  a  huge  impact  on  military  activities  aimed  at  getting 
soldiers,  sailors,  marines,  and  equipment  on  the  shore.  Although  waves  are  generated  in  the  open 
ocean  and  propagate  over  long  distances,  the  last  kilometer  or  less  near  the  shore  is  the  critical 
region  for  landing  and  docking.  Nearshore  wave  transformation  is  solved  in  the  model  STWAVE 
(Smith,  2002)  using  the  wave  action  balance  equation.  The  model  includes  the  processes  of 
shoaling  and  refraction  (increases  in  wave  height  and  changes  in  wave  direction  in  shallow 
water),  wave-current  interaction  (changes  in  wave  properties  caused  by  tidal  or  nearshore 
currents),  wave  growth  due  to  the  wind,  and  wave  breaking.  The  model  is  based  on  linear  wave 
theory  and  does  not  include  reflection  or  bottom  friction.  By  linking  the  nearshore  wave 
transformation  model  STWAVE  with  deepwater  wave  forecasts  or  hindcasts  (at  large  spatial 
scales),  nearshore  waves  can  be  estimated  on  beaches  and  at  harbor  and  inlet  entrances.  Forecasts 


are  used  for  short-term  planning  of  operations  and  hindcasts  are  used  to  gamer  climatologically 
information  for  longer-term  planning  (typical  and  extreme  conditions,  seasonal  variation,  storm 
probability). 

Nearshore  wave  transformation  modeling  in  the  past  has  been  run  on  HPC  platforms  to 
make  forecasts  to  support  war  fighters.  To  meet  stringent  time  requirements  for  producing 
forecasts,  the  model  was  mn  with  very  coarse  spatial  resolution  and  less  frequency  in  time  than  is 
optimal.  Under  the  CHSSI  HFSoLE  project,  STWAVE  was  parallelized  to  exploit  HPC 
capabilities.  Tests  earlier  this  year  demonstrated  that  the  parallel  version  of  STWAVE  was  over 
90  percent  efficient  using  32  processors  on  both  the  Cray  T3E  and  SGI  Origin  3000.  This 
translates  into  increasing  the  model  through-put  by  a  factor  of  28.  The  highly  efficient 
parallelization  allows  improved  resolution  of  the  nearshore  domain  (which  improves  model 
accuracy)  and  more  frequent  model  runs  (more  resolution  in  time),  while  still  meeting  or  beating 
required  forecast  time  constraints.  Improved  nearshore  modeling  times  also  allows  more  time  for 
other  modeling  components  and  for  decision  making.  Figure  4  shows  verification  of  the  parallel 
version  of  STWAVE  at  Duck,  NC,  for  the  year  1997.  The  agreement  with  field  measurements  is 
excellent.  Full-year  simulations  were  not  run  prior  to  the  parallel  capability. 
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Figure  4.  Verification  of  parallel  version  of  STWAVE  wave  heights  at  Duck,  North  Carolina  versus 
observational  data  for  a  one  year  period. 

Continuing  challenges  in  the  work  include  expanding  to  more  HPC  platforms  and  a  larger 
number  of  processors,  implementing  additional  improvements  to  model  efficiency,  and  upgrading 
model  physics.  The  CHSSI  work  will  focus  on  interfacing  STWAVE  with  circulation  and 
sediment  transport  models  to  provide  a  greater  understanding  of  the  physical  processes  that 
impact  directly  on  operations,  equipment,  and  training.  This  work  is  partnered  with  the  Naval 
Research  Lab  as  well  as  with  other  ERDC  researchers.  This  work  falls  within  the  Climate 
Weather  Ocean  CTA. 

Adaptive  Hydraulics  Code  (ADH) 

The  ADaptive  Hydraulics  (ADH)  code  is  a  finite  element  model  constructed  within  the 
Department  of  Defense  (DoD).  ADH  is  designed  to  address  a  host  of  water  related  problems, 
including  resource  protection  and  flow  and  constituent  transport  quantification.  In  this  example 


the  shallow  water  equations  are  being  used.  These  are  the  typical  equation  sets  used  for  flow  in 
rivers  and  tidal  regions.  The  code  uses  a  discrete  Newton  approach  to  develop  the  algebraic 
equation  set.  The  resulting  linear  system  is  solved  using  the  BiCG-Stab  iterative  solver. 
Preconditioners  are  essential  to  make  efficient  computations.  Domain  decompostion 
preconditioners  presently  in  ADH  are,  Jacobi,  1 -level  Additive  Schwarz,  2-level  Additive 
Schwarz,  and  a  2-level  hybrid  scheme.  The  domain  is  broken  into  many  blocks.  The  2-level 
preconditioner  then  performs  two  direct  solves  on  this  aggregate  of  blocks.  The  first  is  a  “fine 
solve”  on  each  individual  block,  this  is  followed  by  a  “coarse  solve”  over  the  entire  domain  but 
with  resulting  factors  representing  “average”  results  per  block.  The  selection  of  an  optimum 
number  of  blocks  for  a  variety  of  numbers  of  processors  can  be  daunting.  In  prior  work,  the  total 
number  of  blocks  resulting  in  near  optimum  timings  appeared  to  be  relatively  constant  over  a 
wide  range  of  numbers  of  processors.  A  series  of  simulations  have  been  performed  in  which 
many  numbers  of  blocks  and  processors  were  chosen  to  try  to  determine  if  this  is  tme.  Runs  were 
made  on  the  ERDC  Origin  03K  and  on  the  ERDC  Cray  T3E.  The  results  for  the  03K  are  shown 
in  Figure  5.  The  investigation  used  2,  4,  8,  and  16  processors  and  up  to  a  total  of  200  blocks. 
The  timings  are  nondimensional  ratios  of  the  best  time  in  the  entire  experiment.  Strictly  speaking 
the  optimum  total  number  of  blocks  is  shown  not  to  be  a  constant.  However,  if  the  optimum  is 
based  upon  the  larger  number  of  processors,  around  80  blocks,  the  times  on  the  other  processors 
would  be  reasonable.  The  results  are  not  terribly  sensitive  except  for  the  lowest  number  of 
processors,  2,  in  which  a  time  about  25%  above  optimum  would  be  found.  Generally  speaking 
the  total  number  of  blocks  required  to  reach  an  optimum  decreases  as  the  number  of  processors 
increases. 


Figure  5.  Total  number  of  blocks  versus  run  time  for  preconditioner  in  ADH.  Timings  based  on  2, 4, 
8,  and  16  processors  on  the  ERDC  Origin  3000. 

Littoral  Sedimentation  and  Optics  Model  (LSOM) 

The  Littoral  Sedimentation  and  Optics  Model  (LSOM)  is  a  derivative  of  the  TRANS98 
model  described  by  Keen  and  Glenn  (1998)  and  Keen  and  Stavn  (2000).  The  bottom  boundary 
layer  component  is  a  modification  of  the  wave-current  interaction  model  of  Glenn  and  Grant 


(1987).  It  calculates  the  combined  shear  stresses  resulting  from  wave  and  current  interaction  in 
the  marine  bottom  boundary  layer.  The  resuspension  and  optical  properties  of  the  sand  and  silt 
component  are  described  by  Keen  and  Stavn  (2000).  The  LSOM  model  has  been  modified  to 
simulate  the  resuspension  of  clay  minerals  following  Gailani  et  al.  (1996).  The  mass  of  sediment 
eroded  from  a  cohesive  bed  is  given  by: 


for  T  >  To,  where  ao  is  a  site-specific  constant,  td  is  time  in  days  since  deposition,  t  is  the  bottom 
shear  stress,  and  the  constants  n  and  m  depend  on  the  depositional  environment.  For  this  study, 
we  do  not  have  sufficient  data  to  calibrate  this  equation  so  we  will  use  values  suggested  by 
Gailani  et  al.  (1996).  This  is  considered  reasonable  because  the  purpose  of  this  study  is  to 
examine  patterns  of  resuspension  and  the  resulting  optical  properties.  The  clays  are  assumed  to 
have  a  mean  floe  diameter  of  10  microns.  This  is  used  in  calculating  their  scattering  coefficients. 

The  LSOM  model  is  run  independently  of  the  hydrodynamic  and  wave  models.  It  uses  a 
time  step  of  3  hours.  Sedimentation  is  simulated  for  the  same  interval  as  the  other  models, 
however.  At  every  model  cell,  the  sand  component  of  the  sediment  is  resuspended  separately  from 
the  clay  component  and  their  optical  characteristics  are  added  to  find  the  total  scattering 
coefficients.  The  Diver  Visibility  DV  is  calculated  from  the  biogenic  and  minerogenic  scattering 
coefficients  using  the  following  relationship 
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where  bewor  is  the  scattering  due  to  chorophyll  (biogenic  particles),  bmi„ero  is  the  scattering  due  to 
resuspended  mineral  sediments,  and  As  is  the  scattering  albedo.  The  scattering  due  to  organic 
particles  is  calculated  from  the  surface  chlorophyll  concentration  derived  from  the  SeaWLFs 
sensor 

=  0.252C‘''*^^(0.635)(0.97) 

where  C  is  the  chlorophyll  concentration. 

The  model  incorporates  MPI  processing  m  calculating  transformation  matrices  for 
input  waves,  currents,  bathymetry,  and  sediment  type.  It  also  uses  MPI  for  the  main 
computation  loop,  assigning  blocks  of  grid  rows  to  individual  processors.  An  efficiency 
of  0.41  has  been  obtained  with  12  processors  on  the  Sun  lOOOOE  (wolfe)  at  NAVO. 

Persian  Gulf  Studies 

Contained  within  the  region  defined  by  the  Persian  Gulf,  Strait  of  Hormuz  and  the  Gulf 
of  Oman  is  one  of  the  busiest  and  most  important  shipping  lanes  in  the  world.  A  vessel  passes 
through  the  Strait  of  Hormuz  every  6  min  and  approximately  60%  of  the  world’s  marine  transport 
of  oil  comes  from  this  region  (Reynolds,  1993).  For  a  region  of  such  economic  importance, 
comprehensive  studies  investigating  the  circulation  of  the  region  have  been  sparse,  in  part,  due  to 
the  scarcity  of  observations. 

The  Persian  Gulf  is  a  semi-enclosed  marginal  sea  located  at  the  southeastern  end  of  the 
Arabian  Peninsula.  Moving  from  the  SW  coast  counterclockwise  along  the  SE  coast.  Gulf  waters 
are  bordered  by  the  United  Arab  Emirates,  Qatar,  Bahrain,  Saudi  Arabia,  Kuwait,  Iraq  and  Iran. 
Depths  in  the  Arabian  Gulf  are  quite  shallow  averaging  approximately  50  m.  Connection  to  the 
Gulf  of  Oman  and  the  eastern  Indian  Ocean  is  through  the  Strait  of  Hormuz  whose  narrow 
constriction  controls  the  tidal  character  of  the  basin,  separating  it  from  the  Indian  Ocean  co-tidal 
system.  The  bathymetry  of  the  Arabian  Gulf  is  markedly  asymmetrical  about  the  NW-SE  axis  of 


the  basin.  A  deep  channel  off  the  Iranian  coast  contrasts  the  shallow  broad  shelf  on  the  Arabian 
side  of  the  Gulf 

The  following  section  describes  a  series  of  models  that  are  utilizing  Challenge 
computational  resomces  to  study  waves,  tides  and  sediment  transport  in  the  Persian  Gulf 

ADCIRC 

The  tide  and  storm  surge  computations  in  the  Persian  Gulf  are  made  using  the  fully 
nonlinear,  two-dimensional,  barotropic  hydrodynamic  model  ADCIRC-2DDI  (Luettich  et  al., 
1992).  The  ADCIRC  model  has  an  extensive  and  successful  history  of  tidal  prediction  in  coastal 
waters  and  marginal  seas  (e.g.  Westerink  et  al,  1994;  Blain  et  al.,  1994;  Luettich  and  Westerink, 
1995;  Blain,  1998;  Fortunato,  1998;  Luettich  et  al.,  1999).  The  model  is  well  suited  to  tidal 
studies  because  of  its  computational  efficiency  that  is  due  in  part  to  implementation  of  the  finite 
element  method,  MPI  parallelization  of  the  code,  and  the  use  of  an  iterative  sparse  matrix  solver. 
One  advantage  of  the  model  for  tidal  analyses  is  the  facility  for  timely  and  complete  run-time 
harmonic  decomposition  for  astronomical  and  nonlinear  compound  and  over-tide  components  of 
the  tidal  signal.  One  can  expect  to  complete  simulations  which  extend  from  months  to  a  year  and 
employ  30  second  time  step  increments  expediently  on  both  workstation  and  supercomputer 
platforms  without  unreasonable  storage  requirements.  The  ADCIRC  model  configuration  for  the 
Persian  Gulf  is  similar  to  the  operational  model  at  NAVOCEANO  described  in  Blain  et  al. 
(2002). 

The  ADCIRC-2DDI  model  is  based  on  the  well-known  shallow  water  equations  that  are 
derived  through  a  vertical  integration  over  the  water  column  of  the  three-dimensional  mass  and 
momentum  balance  equations  subject  to  the  hydrostatic  assumption  and  the  Boussinesq 
approximation.  Though  surface  heating  and  evaporative  fluxes  together  are  known  to  be 
dominant  mechanisms  for  inducing  circulation  in  the  Arabian  Gulf  (e.g.  Chao  et  al,  1992), 
baroclinic  processes  are  neglected  here  leaving  only  the  tide  and  surge  component  of  the 
circulation.  Bottom  stress  terms  are  parameterized  using  the  standard  quadratic  friction  law  and 
no  lateral  mixing  due  to  diffusion  or  dispersion  is  considered.  Numerical  solution  of  the  ADCIRC 
model  governing  equations  is  achieved  by  recasting  the  continuity  equation  into  a  generalized 
wave  continuity  equation  (Lynch  and  Gray,  1979;  Kinnmark,  1984).  The  discrete  problem  in 
space  is  approached  through  an  application  of  the  finite  element  method.  The  wetting  and  drying 
of  computational  elements  is  possible  within  ADCIRC-2DDI  but  is  not  activated  for  these 
computations. 

The  source  of  bathymetry  used  with  the  ADCIRC  model  is  the  DBDB-V  2  minute  data 
base  (Naval  Oceanographic  Office,  1997)  on  which  a  minimum  depth  of  3  m  is  imposed  to 
eliminate  drying  of  computational  points  along  the  shoreline.  The  model  domain  includes  the 
Persian  Gulf  Strait  of  Hormuz,  and  extends  to  approximately  the  200  m  isobath  in  the  Gulf  of 
Oman.  The  finite  element  mesh  for  this  domain  contains  17440  computational  points  whose 
spacing  ranges  from  0.5-1.0  km  near  the  Shatt  al  Arab  river  inflow  in  the  NW  to  8  km  in  the  Gulf 
of  Oman. 

Tidal  forcing  from  8  constituents  (Ql,  01,  Kl,  N2,  M2,  S2,  K2,  and  PI)  is  applied  both 
at  the  open  boundary  and  internal  to  the  domain  through  the  tidal  potential.  Values  for  tidal 
constituents  prescribed  at  the  open  boundary  are  obtained  from  results  of  the  Grenoble  global  tide 
model  (LeProvost  et  al.,  1994).  Wind  velocities  at  10-m  over  a  27  km  resolution  grid  are  obtained 
from  the  Navy’s  COAMPS  (Hoder  and  Doyle,  1999)  model.  Wind  stresses  that  serve  as  surface 
forcing  for  the  ADCIRC  model  are  then  computed  using  the  formulation  of  Garratt  (1977).  The 
bottom  fnction  coefficient  is  imiform  and  set  equal  to  0.0015.  Simulations  are  spun  up  from  a 
homogeneous  initial  condition  using  a  ramp  in  time  equivalent  to  fifteen  days.  TTie  model  run 
extends  30  days  beyond  the  spin-up  period  (i.e.  June  1,  2001  to  June  30,  2001)  during  which  sea 
surface  elevation  and  currents  for  all  grid  points  are  written  every  two  hours.  A  time  step  of  30 


seconds  is  more  than  sufficient  to  satisfy  the  Courant  criterion  over  the  entire  domain.  The 
ADCIRC  model  of  the  Persian  Gulf  ran  on  4  processors  of  the  IBM  SP3  (375  MHz)  in  3:20 
hours.  Figure  6  depicts  ADCIRC  water  elevation  and  currents  for  June  1,  2001  at  00  GMT. 


Figure  6.  ADCIRC  Arabian  Gulf  simulation  results  for  June  1, 2001.  Color  shading 
indcates  water  elevation  (m),  arrows  show  direction  of  flow. 


SWAN 


A  series  of  wave  simulations  were  performed  in  the  Persian  Gulf  using  the  OpenMP 
version  of  the  Simulating  Waves  Nearshore  (SWAN)  wave  model.  The  serial  code  for  SWAN 
was  parallelized  by  the  Mississippi  State  University  and  NAVO  MSRC  Programming 
Environment  and  Training  (PET)  Office  located  at  Stennis  Space  Center,  MS.  SWAN  was 
originally  a  serial  code  that  was  considered  unamenable  to  parallelization.  Due  to  the  data 
dependencies  of  the  implicit  technique,  a  straight  forward  domain  decomposition  of  the  spatial 
grid  was  not  possible  without  completely  altering  the  numerical  algorithm.  Instead,  a  shared 
memory  pipelined  parallel  technique  was  adopted  that  would  require  no  changes  in  numerical 
scheme  or  data  layout.  By  introducing  OpenMP  directives  at  a  fairly  high  level  in  the  numerical 
scheme,  the  parallel  version  of  SWAN  runs  with  the  same  input  and  output  files  as  the  serial 
code. 

SWAN  (Booij  et  al,  1999)  is  based  on  the  wave  energy  balance  equation  with  source  and 
sink  terms.  Boundary  conditions  were  provided  by  directional  wave  spectra  obtained  by  running  a 
nested-version  of  the  OpenMP  deep-water  WAM  wave  model.  The  WAM  spectra  were  specified 
on  the  SWAN  eastern  boundary  at  four  locations  along  58.0  “E.  SWAN  was  forced  with 
COAMPS  10-m  winds  for  two  simulation  periods:  a)  September  1-30,  2000  and  b)  June  1-30, 
2001.  The  model  was  run  in  a  non-stationary  (time-dependant)  mode  with  a  15-minute  timestep. 
For  these  preliminary  studies,  tides  were  not  considered.  Figure  7  depicts  the  SWAN  wave  height 
and  direction  for  June  1, 2001.  Note  the  high  waves  in  the  northwest  portion  of  the  Persian  Gulf 
due  to  strong  wind  forcing.  Future  work  will  investigate  the  use  of  ADCIRC  tides  as  input  to 
SWAN.  Wave  heights  and  directions  from  SWAN  are  used  as  input  to  the  Littoral  and 
Sedimentation  Optics  Model  (LSOM)  described  in  the  next  section. 
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Figure  7.  SWAN  wave  height  (ft)  for  June  1, 2001  00  GMT.  Arrows  depict  direction  waves  are 
moving  toward. 

LSOM 

The  bottom  sediment  in  the  Persian  Gulf  consists  primarily  of  carbonate  (calcite  and 
aragonite)  and  evaporite  sediments,  including  halite,  gypsum,  and  anhydrite,  which  are  found  in 
the  sabkha  areas  along  the  coast  (Ross  and  Stoffers,  1978).  Wind-blown  terrigeneous  sediments 
comprise  less  than  one-third  of  the  sediment  within  the  area  and  river  inflow  is  minitnal  The 
majority  of  the  sediment  is  silt  and  the  sand  component  is  generally  less  than  20%. 

This  study  simulates  resuspension  of  the  sediments  within  the  Persian  Gulf  using  LSOM. 
The  sedimentation  model  is  run  on  a  Cartesian  grid  with  horizontal  dimensions  of  3262  m  and 
3702  m  along  the  east-west  and  north-south  axes,  respectively.  The  bottom  boundary  layer  is 
represented  by  3 1  sigma  levels,  with  10  located  within  the  wave  boundary  layer.  The  sediment 
within  the  Persian  Gulf  is  represented  by  20  size  classes  with  a  mean  of  62  microns.  The  resulting 
predicted  diver  visibility  will  be  compared  to  the  value  calculated  directly  from  the  satellite 
sensor  (Figure  8a).  Figure  8  shows  the  relationship  between  low  diver  visibility  and  resuspended 
sediment  in  shallow  water  near  the  coast,  as  predicted  by  LSOM  (Figure  8b).  Note  the  high 
visibility  within  the  embayments  to  the  north  and  south  of  the  Qatar  peninsula,  as  indicated  by 
yellow  hues  in  Figure  8a  and  blues  in  Figure  8b. 


Figure  8.  Diver  visibility  for  the  Persian  Gulf  on  20  June  2001:  (a)  computed  from  the 
SeaWiFS  sensor  and  (b)  calculated  from  LSOM  using  resuspended  sediment  and  constant 
chlorophyll. 


The  model-predicted  diver  visibility  for  the  second  half  of  June  2001  (Figure  9)  shows 
daily  variability  related  to  tides  as  well  as  larger  changes  associated  with  storms.  The  pattern  of 
predicted  visibility  is  correlated  with  wind  waves  generated  within  the  Persian  Gulf. 

Time  series  of  Diver  VMbIHtjf  for 
Idobriiig  located  at  (si 25.6332) 


Adriatic  Sea  Studies 


Researchers  at  NRL-Monterey,  NRL-Stennis,  and  the  NATO  SACLANT  Undersea 
Research  Centre  are  using  HPC  resources  to  develop  and  study  the  performance  of  a  high 
resolution  coupled  wind/current/wave/sediment  coastal  modeling  system.  The  wind  is  provided 
by  CO  AMPS,  triple-nested  with  a  4  km  resolution  over  the  entire  Adriatic.  The  ocean  currents  are 
computed  by  NCOM  at  a  2  km  resolution.  The  waves  are  computed  by  SWAN  on  the  same  2  km 
grid.  The  LSOM  model  will  be  driven  using  the  currents  and  waves  from  NCOM  and  SWAN  to 
provide  simulations  of  sediment-related  optical  clarity  for  use  such  as  diver  visibility  and  lidar 
penetration.  These  models  will  be  critically  assessed  during  a  large  field  program  in  the  Adriatic 
Sea  that  will  be  carried  out  from  Sep  2002-May  2003. 

Figure  10  shows  a  snapshot  of  a  typical  “Bora”,  a  frequent  winter  meteorological  event 
where  strong  winds  blow  across  the  Northern  Adriatic.  The  currents  drive  the  fresh-water  plume 
from  the  Po  River  down  the  coast,  and  also  generate  waves  capable  of  resuspending  sediments 
deposited  near  the  Po  River  mouth. 

A  significant  results  so  far  is  that  higher-resolution  nested  wind  modeling  made  possible 
by  HPC  resources  has  for  the  first  time  produced  wave  fields  that  compare  well  with  observed 
buoy  data  without  the  application  of  any  scale  factors  in  this  region.  Previous  lower  resolution 
meteorological  products  had  to  be  multiplied  by  factors  of  1.3  or  1.5  before  they  produced 
acceptable  results. 

These  models  use  the  HPC  systems  as  follows.  The  OpenMP  version  of  SWAN  is  being 
used  on  the  SGI  Ruby  (8-16  processors)  under  the  HFSoLE  Grand  Challenge.  NCOM,  using 
MPI,  is  run  typically  with  16  nodes  on  the  ERDC  03K  (ruby).  Both  the  NCOM  and  COAMPS 
runs  were  performed  under  the  Coupled  Mesoscale  Modeling  of  the  Atmosphere  and  Ocean  (379) 
Grand  Challenge. 
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Figure  10.  Snapshot  of  a  "Bora"  in  the  Northern  Adriatic,  a)  COAMPS  wind  stress,  b)  NCOM 
salinity  (PPT),  c)  SWAN  wave  height  m),  and  d)  NCOM  2-m  currents  (m/s). 


Model  Coupling  Environmental  Library  (MCEL) 


In  the  realm  of  geophysical  modeling  the  current  state-of-the-art  models  have  the 
capability  to  run  at  very  high  spatial  resolutions.  This  has  led  to  a  drastic  increase  in  the  accuracy 
of  the  physics  being  predicted.  Due  to  the  increased  numerical  accuracy,  once  neglected  effects 
such  as  non-linear  feedback  between  different  physical  processes  can  no  longer  be  ignored.  The 
ocean*s  deep  water  circulation,  surface  gravity  waves  and  atmosphere  above  can  no  longer  be 
treated  as  independent  entities,  and  must  be  considered  as  a  coupled  system.  One  solution  to  this 
problem  is  to  link  models  together  through  a  series  of  surface  variables.  An  example  would  be 
the  evaporative  cooling  of  the  ocean,  which  at  a  simple  level,  requires  the  sea  surface 
temperature,  humidity  and  temperature  of  the  atmosphere  and  in  turn  would  return  the  mass  and 
heat  flux  into  the  atmosphere. 

Under  the  High  Fidelity  Simulation  of  Littoral  Environments  (HFSoLE)  project  the 
Model  Coupling  Environmental  Library  (MCEL)  is  under  development.  The  MCEL 
infrastructure  utilizes  a  data  flow  approach  where  coupling  information  is  stored  in  a  single  server 
or  multiple  centralized  servers  and  flows  through  processing  routines  called  filters  to  the 
numerical  models  which  represent  the  clients.  These  filters  represent  a  level  of  abstraction  for  the 
physical  or  numerical  processes  which  join  together  different  numerical  models.  The  extraction 
of  the  processes  unique  to  model  coupling  into  independent  filters  allows  for  code  reuse  for  many 
different  models.  The  communication  between  these  objects  is  handled  by  the  Common  Object 
Request  Broker  Architecture  (CORBA).  In  this  paradigm  the  flow  of  information  is  fUlly 
controlled  by  the  clients.  Figure  1 1  represents  a  hypothetical  example  of  how  such  a  system 
might  be  used. 
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Figure  11.  Example  of  how  MCEL  can  be  used  to  provide  data  between  HFSoLE  models. 


In  this  example  every  box  represents  its  own  process,  either  numerical  model,  filter  or  the  central 
server.  This  allows  for  an  MDMP  parallelization  of  the  different  models.  Each  model  has  its 
own  pool  of  CPUs  to  work  from  which  provides  greater  computational  efficiency.  This  example 
was  implemented  for  verification  of  the  coupling  scheme  and  it  was  shown  that  it  provided  a 
equivalent  results  to  the  traditional  file  I/O  coupling  scheme  in  half  the  time  on  a  set  of  Linux 
workstations. 
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